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This  paper  proves  that  a  large  class  of  iterative  schemes  can  be  used  to 
solve  a  certain  constrained  minimization  problem.  The  constrained 
minimization  problem  considered  involves  the  minimization  of  a  quadratic 
functional  subject  to  linear  equality  constraints.  Among  this  class  of 
convergent  iterative  schemes  are  generalizations  of  the  relaxed  Jacobi,  Gauss- 


Seidel,  and  symmetric  Gauss-Seidel  schemes./!^ 
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SIGNIFICANCE  AND  EXPLANATION 


Consider  the  following  problem:  Find  the  real  n-vector  which 

1  T  T 

minimizes  f(x)  =  y2X  Ax  -  x  r  subject  to  the  constraints 
T 

g(x)H£x-SKO.  The  purpose  of  this  paper  is  to  show  that  there  is  a 
large  class  of  iterative  schemes  which  can  be  used  to  solve  such  problems* 
These  schemes  are  particularly  effective  when  the  matrix  A  is  large  and 
sparse  and  there  are  only  a  moderate  number  of  constraints  7  one  example  of 
such  a  problem  is  described  at  the  end  of  this  paper*  / 


THE  NUMBRICAL  SOLUTION  OP  A  CLASS  OF 
CONSTRAINBD  NININIZATZON  PROBLEMS 


Nira  Dyn  and  Warren  E*  Ferguson 

t 1 •  Introduction 

In  this  paper  %re  will  present  several  iterative  schenws  which  solve  the  following 
constrained  mininization  problem: 

Problem  1:  Find  the  real  n-vector  which  minimizes  f(x)  =  x'^Ax  -  x*^r 

T 

subject  to  the  constraints  9(x)2ex-s*0* 

Here  A  is  a  real  syimnetric  nonnegative  definite  n  x  n  matrix,  E  is  a  real  n  ^  m 
matrix  with  full  column  rank,  r  is  a  real  n*-vector,  and  s  is  a  real  m-vector* 

As  discussed  in  section  2,  the  theory  of  quadratic  programming  [Hadley]  states  that 
under  reasonable  conditions  on  A  and  E  the  solution  of  Problem  1  exists  and  is 
unique.  Furthermore,  under  these  conditions  on  A  and  E  the  solution  x«  of  Problem  1 
is  the  x  component  of  the  solution  (x^,X^)  of  the  following  problem: 

Problem  2t  Find  the  real  n-vector  x«  and  the  real  m-vector  which 

solves  the  linear  system 


In  section  3  %re  describe  the  convergence  of  a  large  class  of  iterative  schemes  used  to 
solve  Problem  2,  and  hence  Problem  1.  Although  our  iterative  schemes  are  generally 
applicable  to  these  problems,  they  are  typically  efficient  only  when  A  is  a  large  sparse 
matrix  and  there  are  only  a  moderate  number  of  constraints.  Zn  this  situation  the  usual 
methods  used  to  solve  these  problems  become  inefficient.  The  application  of  similar 
iterative  schemes  to  the  minimization  of  a  quadratic  form  under  inequality  constraints  is 
investigated  in  [Mangasarian] . 


School  of  Mathematical  Sciences,  Tel-Aviv  university,  Israel. 

•• 

Department  of  Mathematics,  Southern  Methodist  University,  Dellas,  Texas  75275. 


Sponsored  by  the  United  States  Army  under  Contract  Ho.  DAAG29-80-C-0041 . 


Our  uork  was  aotivut^d  by  thm  work  of  [Toblor]  in  which  «  wnrinnt  of  ono  of  thn 
Iterntivo  schnnnn  dnncribod  in  this  pnpor  %iss  used  to  nuMrically  construct  a  awooth 
surfaca  froai  a99ra9atsd  data*  This  application  is  dsscribsd  in  suction  4*  Thu  nunsrioaX 
solution  of  ProbluBS  1  and  2  has  also  buun  considarud  by  othar  authors;  in  particular  wa 
raantlon  tha  work  prasantad  by  [Luanbargar] ,  [PaigSf  Saundars)*  and  [Gill#  Murray]* 


$2*  Preliminaries 

In  the  previous  section  we  stated  that  under  reasonable  conditions  on  A  and  B  the 
solution  X*  of  Probleai  1  is  part  of  the  solution  of  Problem  2*  This  statement 

is  contained  in  the  following  theorem. 

Theorem  2.1:  Assume  that 

(a)  A  is  a  real  symmetric  nonnegative  definite  matriXf 

(h)  E  is  a  real  matrix  with  full  column  rank,  and 
T 

(c>  A  and  E  have  no  nontrivial  null  vectors  in  common* 

Then  the  solutions  of  Problems  1  and  2  exist  and  are  unique. 

Furthermore:  if  x«  is  the  solution  of  Problem  1  then 
(X*,  (K^E)”'E'^(r  -  Ax*))  is  the  solution  of  Problem  2, 
if  tx*»X^)  is  the  solution  of  Problem  2  then  x*  is  the 
solution  of  Problem  1 • 

Proof:  oee  the  treatment  of  quadratic  programming  given  in  {Hadley). 

■ 

Corollary  2.2:  Under  the  assumptions  of  Theorem  2.1  the  matrix 


Is  nonsingular. 

Proof:  Observe  that  this  matrix  is  the  coefficient  matrix  of  the  linear  system  in  Problem 
2.  Under  the  assumptions  of  Theorem  2.1  this  linear  system  has  only  unique  solutions, 
therefore  as  sho%m  in  [Stewart)  the  coefficient  matrix  is  nonsingular. 

■ 

The  iterative  schemes  we  use  to  solve  Problem  2  are  all  based  upon  a  splitting 

A  -  fi  *  C 

of  the  matrix  A,  and  they  have  the  following  form.  Given  an  initial  iterate  (>^q*X^) 
define  ^  ”  1«2,3,...  to  be  the  solution  of  the  linear  system 
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Of  course,  for  this  linear  stationary  iterative  method  to  be  well  defined  it  is  necessary 
and  sufficient  that  the  matrix 


be  nonsingular*  This  problem  is  addressed  by  the  following  theorem* 
llieorem  2*3:  In  addition  to  the  assumptions  of  Theorem  2*1  let 


(d)  A  -  B  -  C  , 

(e)  B  be  a  real  nonsingular  matrix,  and 

(f)  2A'»'C<fc'^  bea  positive  definite  matrix* 
Then  the  iterative  scheme  (1)  is  well^efined* 

Proof:  Since  B  is  nonsingular  then 


B 


0 


b“^e 


T  - 
-E  B 


{? 


I 


therefore  it  follows  that  the  matrix  on  the  left  hand  side  of  the  equality  is  nonsingular 
if  and  only  if  E^b“^B  is  nonsingular*  To  prove  that  is  nonsingular  let  us  prove 

that  has  no  nontrivial  null  vector*  If  E^B  ^El  «  0  then 

0  -  X^E^B*’eA 

-  (b“’eX)^b^(b“^eX) 

-  V2  +  B^)(b"^EX) 

-  V2  ♦  C  +  C^)(B*^EX)  . 

This  Implies  B*^eX  m  0,  since  2A  <f  C  4-  is  a  positive  definite  matrix,  and  so  X  «  0 


because  B  is  a  matrix  with  full  column  rank 


f 

i. 

L«t  US  now  <)o8crib«  ons  procedure  for  solving  the  llneer  system  (1)  for 
Step  It  Solve 

I 

cor 

Step  2 :  Solve 

(E  B  *  *  ’^Ic  *  * 

for  1. . 
k 

Step  3 1  Solve 
“‘*k  - 

Kjj  -  ^  - 

Step  2  requires  the  solution  of  a  full  linear  systen  of  order  Therefore  the 
Iterative  scheee  is  efficient  only  when  m  «  n* 

In  the  next  section  we  will  see  that  if  assus^ion  (f)  of  Theorem  2.3  is  slightly 


strengthened  then  the  iterative  scheme  (1)  is  not  only  well-defined  but  also  convergent 


|3.  Convergence  of  the  Iterative  Scheeea 


One  set  of  conditions  which  guarantees  the  convergence  of  the  Iterative  scheM  (1)  is 
described  in  the  following  theorea. 

Theorea  3« 1 :  Assume  that 

<a>  A  is  a  real  syanetric  nonnegative  definite  matrix# 

(b)  £  is  a  real  aatrix  with  full  column  rank, 

*r 

(c)  A  and  E  have  no  nontrivial  null  vectors  in  common# 

(d)  A  «  B  «  C  # 


(e)  B  is  a  real  nonsingular  matrix#  and 

(f)  A  'f  C  'f  is  a  positive  definite  matrix* 

then  the  Iterative  scheme  (1)  is  well-defined  and  convergent* 

Proofs  From  Theorem  2*1  we  deduce  that  a  solution  of  Problem  2  exists  and  is  unique* 
Since  A  is  a  nonnegative  definite  matrix  and  A  4  C  is  a  positive  definite  matrix 

then  2A  C  is  a  positive  definite  matrix*  From  Theorem  2*3  we  therefore  deduce 

that  the  iterative  scheme  (1)  is  %iell-deflned* 

As  shown  in  [Wendroff]#  the  iterative  scheme  (1)  is  convergent  if  and  only  if  each 
eigenvalue  of  the  matrix 


* 
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has  magnitude  less  than  one*  z«t  us  therefore  show  that  if  y  is  a  nonsero  eigenvalue  of 
the  matrix  12)  then  the  magnitude  of  y  is  less  than  one* 

Since  y  is  an  eigenvalue  of  the  matrix  (2)  then  there  are  complex  vectors  u#v  not 
both  sero  for  %rhich 
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(3) 


Let  us  now  argue  that  u  ^  0  and  u*^Au  >  0*  Since 
nonnegative  definite  matrix  then  clearly  u^Au  >  0#  and 


A  is  a  real  sysiietrlc 
u^Au  ■  0  only  if  Au  •  0* 


•6“ 


•* 


I. 


* 


i 


However  Au  «  0  only  if  u  •  0#  for  (3)  states  tnat  B  u  «  0  and  by  hypothesis  A  and 

1* 

E  have  no  nontrivial  null  vectors  in  connon*  But  u  *  0  only  if  v  ■  0,  for  by 
hypothesis  E  has  full  oolunn  rank  and  (3)  Inplies  that  Bv  «  0  when  u  «  0«  Since 
u»v  are  not  both  zero  then  we  conclude  u  0  and  u^Au  >  0. 

Let  us  now  establish  the  fact  that 

{l  -  ||i|*}u“au  -  |l  -  n|*u**(A  +  C  +  C*)u  .  (4) 

he  bevin  with  the  identity 

u**Au  -  (Mu)^A<|iu)  •  (U  -  |iu)^A(u  -  Jiu)  2Re((u  -  llu)^A(llu)}  •  (5) 

Using  (3)«  and  the  fact  that  A  »  B  -  C«  we  find  that 

(u  -  Mu)**A(|iu)  .  (u  -  |iu)‘*(B-C)(Uu) 

«  (u  -  |iu)^(|ffio  -  MCu) 

»  (u  -  im)^<Cu  -  uBv  -  |iCu) 

«  (u  -  Uu)**C{u  -  iio)  , 
which  reduces  (5)  to  the  result  stated  in  (4)* 

By  hypothesis r  A  ^  C  *  is  a  positive  definite  aatrix*  Since  u  ff  0  we  know 

that  u^(A  *  C  C^)u  >  0^  and  so  (4)  Inplies  that  either  jv)  <  1  or  ii  «  1*  However  it 

is  inpossible  that  p  «  1,  for  if  M  *  1  then  (3)  would  inply  that  u^v  are  both  zero 

because  by  Corollary  2«2  the  natrix 

is  nonsingular* 

Let  us  now  describe  several  iterative  schenes  whose  convergence  is  assured  by  Theoren 

3*1. 

Corollary  3*2t  Assune  that 

(a)  A  is  a  real  symetric  nonnegative  definite  natrix# 

(b)  B  is  a  real  natrix  with  full  colunn  rairic# 

(e)  A  and  B^  have  no  nontrivial  null  vectors  in  oonsKm. 


(g)  A«D-Xi*L^  wh«r«  D  is  a  nonsingular  diagonal  matrix 
and  L  is  a  strictly  lower  triangular  matrix* 

Then  the  iterative  scheme  ( 1 )  is  convergent  for  the  following  choices 

of  B  and  C* 

1  1H()  T 

+  L  +  L  with  w  >  0  chosen  so  small  that 

2 

^  D  -  A  is  a  positive  definite  matrix* 

(2)  B-jD-I„C»  D  +  with  0  <  w  <  2, 

2-u  -11  -11  T 

(3)  B  -  (ijp)  -  h)D  ’(io  -  U\ 

2— —1  1— (1>  >1  1— U  T 

C  -  (— )  (—  D  +  L)D  i~  0  +  I*)  with  0  <  w  <  2. 

Proof:  It  is  clear  that  assumptions  (a)  thru  (e)  of  Theorem  3.1  are  valid  for  each  of  the 

above  choices  for  B  and  C«  Therefore  the  iterative  scheme  (1)  will  be  convergent  if 

assumption  (f)  of  Theorem  3*1  is  valid  for  each  of  the  above  choices  of  B  and  c*  For 

the  first  choice  of  B  and  c  we  find  that 

T  2 

A  +  C  +  C^--D-A 

and  so  assun^tion  (f)  of  Theorem  3.1  is  valid  if  w  >  0  is  chosen  so  small  that  0  -  A 

(0 

is  positive  definite.  For  the  second  choice  of  B  and  C  we  find  that 

A  +  C  +  C^-^0 

and  so  assus^tion  (f)  of  Theorem  3.1  is  valid  if  0  <  m  <  2.  For  the  third  choice  of  B 
and  C  we  note  that 

A  +  C  +  C'’^-B  +  C  , 

where  B<C)  is  a  real  symmetric  positive  (non-negative)  definite  matrix  if  0  <  «  <  2, 
and  so  assumption  (f)  of  Theorem  3.1  is  valid  if  0  <  u  <  2* 


We  note  that  the  firsts  second,  and  third  choices  of  B  and  c  described  in 
Corollary  3*2  correspond  respectively  to  the  usual  JOR,  SOR,  and  S80R  splittings  of  A 
described  in  [Young] •  Under  further  assumptions  Corollary  3*2  can  be  extended  to  the  line 
and  block  versions  of  JOR,  SOR,  and  SSOR*  Furthermore,  there  is  an  obvious  generalisation 
of  Theorem  3*1  to  complex  matrices* 


}4«  Application  of  the  Iterative  Scheme 

Let  us  consider  the  problem  of  estimating  a  certain  geographically  varying  quantity 
over  a  bounded  geographical  region ^  given  the  integrals  of  the  quantity  over  several 
disjoint  sub^regions* 

Let  represent  this  finite  geographical  region  and  Q.  its  disjoint  sub- 

12  m 

regions.  In  addition  let  v(x,y)  represent  the  function  over  ft  whose  values  we  wish  to 
estimate. 

We  shall  choose  the  function  u*(x,y)  by  requiring  that  it  minimize  the  functional 

J(u)  “V2/JJ  lVu(x,y)  l^dxdy  (5) 

among  all  functions  in  the  Sobolev  space  H^ft)  which  satisfy  the  constraints 

u(x,y)dxdy  *  s^  for  i  ■  1,2, ...fm  •  (6) 

i 

Here  the  numbers  ^irS2r-**«s^  are  the  given  values  of  the  integral  of  v(x,y)  over  the 

subregions  ft,#ft. f...#ft  •  This  problem  was  considered  by  [Tobler]  and  analyzed  in  [Dyn, 
12  m 

Wahba] 2  in  particular  the  paper  by  [Dyn,  Wahba]  demonstrates  that  this  problem  is  well- 
posed. 

This  problem  is  discretized  by  the  finite  element  method  as  described  in  [Friedrichs, 

Keller].  Here  a  triangulation  ft^  of  ft  is  introduced  and  u*(x,y)  is  approximated  by 

the  continuous  function  u(x,y)  which  is  a  linear  function  of  x  and  y  in  each  triangle 

of  ft  ,  minimizes  the  functional  (5),  and  satisfies  the  constraints  (6). 

T 

Suppose  that  the  vertices  P^,p2#...fpQ  of  the  triangles  in  ft^  are  labeled  in  some 
order.  If  the  i-th  component  of  the  n- vector  u  represeits;  the  value  of  u(x,y)  at  P^^ 
then  the  functional  (5)  admits  that  representation 

J(u)  -t^u^Au 

where  A  is  a  real  symmetric  nonnegative  definite  n  n  matrix  having  positive  diagonal 
entries.  In  addition  the  constraints  (6)  admit  the  representation 

•  s 

where  E  is  a  real  n  m  matrix  and  s  is  a  real  m-vector  whose  i-th  component  is 
s^.  If  the  triangulation  ft^  is  sufficiently  fine  then  the  matrix  B  will  have  full 


k 
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colunn  rank*  Therefora  tha  n^vactor  u  can  ba  dataminad  by  any  of  tha  itaratlva  achaiaaa 
descrlbad  in  Corollary  3*2* 

Tha  convarganca  of  tha  aolutlon  u  to  tha  solution  u*  of  problan  (5)  and  (6),  as 
tha  trian9ulation  bacoaa  flnar«  is  daitonstratad  in  [Dyn,  Farguson] • 
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